Speeding-up one-versus-all training for extreme classification via mean-separating initialization

Abstract In this paper, we show that a simple, data dependent way of setting the initial vector can be used to substantially speed up training linear one-versus-all classifiers in extreme multi-label classification (XMC). We discuss problem choosing weights from perspective three goals. want start region weight space (a) with low loss value, (b) is favourable for second-order optimization, and (c) where conjugate-gradient (CG) calculations performed quickly. For margin losses, such an initialization achieved by selecting it separates mean all positive (relevant label) instances negatives – two quantities calculated quickly highly imbalanced binary problems occurring XMC. demonstrate speedup $$5\times$$ 5 × on Amazon-670K dataset 670,000 labels. This comes part reduced number iterations need due starting closer solution, implicit negative-mining effect allows ignore easy CG step. Because convex nature optimization problem, without any degradation accuracy. The implementation found at https://github.com/xmc-aalto/dismecpp .